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Implementing a Handheld Program 

Lessons from a District-Level Initiative 


The principal investigator of this handheld program shares the lessons his team 
learned about teacher professional development and integration of new technologies. 


E xcitement about handhelds must be tem- 
pered by an understanding of what is typi- 
cally required when any new technology is 
introduced into classrooms. We know, for example, 
that to succeed, handheld initiatives will need clear 
educational objectives and a logical theory of action 
that guides implementation. We can anticipate that 
teachers will need opportunities to learn how to use 
handheld computers if they are to integrate them 
successfully into instruction. Teachers will also need 
ongoing technical and pedagogical support to sustain 
changes they make to their instruction over time. 

Although these particular requirements are not 
unique to handheld computers, my group s work at 
SRI International to support and study a district- 
level initiative called Project WHIRL (Wireless 
Handhelds for Improving Reflection on Learning) 
has identifled some ways these needs are shaped by 
the particular capabilities and limitations of these 
devices. This article explores what we have learned 
about the opportunities and challenges associated 
with implementing a district-level project with hand- 
held computers. 

Project WHIRL 

Project WHIRL began with the formation of a part- 
nership between researchers at SRI s Center for Tech- 
nology in Learning and school administrators at the 
Beaufort County School District (BCSD) in South 
Carolina. In 2001, SRI and BCSD applied for and 
won a three-year grant from the National Science 
Foundation to investigate the potential of handheld 
computers to support more effective classroom assess- 
ment in middle-grade science classrooms. We hoped 
to take advantage of handheld computers’ portability 
and their data capture and aggregation capabilities to 
make assessment easier for teachers to do frequently. 
We did not intend just to develop tests for handheld 
computers, however. Instead, we hoped to provide 
teachers with tools they could use in an everyday 
fashion that would allow students to be actively in- 
volved in monitoring their own learning and that 


would promote conversations about what students 
are mastering and struggling with in the classroom. 
We achieved this goal in part thanks to our design 
process: last year, teachers used tools in their science 
classes one out of every four days. The tools support- 
ed a variety of assessment purposes as well, helping 
teachers and students alike understand what students 
can know and do. 

A key feature of Project WHIRL was the involve- 
ment of teachers in designing the software they 
would use for classroom assessment. We decided to 
include teachers in the design process from start to 
finish, in an effort to increase the likelihood that the 
software developed would be usable by them and 
adaptable to real-world classroom contexts. Each 
teacher belonged to one of three design teams, which 
consisted of a facilitator, two teachers, a software de- 
veloper, and assessment researchers from SRI. Each 
team was responsible for developing one or more 
software programs for use in the classroom. The fa- 
cilitators sought teachers’ ideas first, as a matter of 
principle, before sharing their own ideas or suggest- 
ing activities to the team. The process proved very 
successful: by the end of the 2002-03 school year, 
teachers who had participated in design teams were 
fluent in the use of the software and became cham- 
pions for their software because it met critical needs 
faced by science teachers in real classrooms. Some of 
this software is now available at the project Web site 
(http : / / WWW. p ro j ectwhirl . or g/) . 

In the 2003-04 school year, the project conducted 
a held trial of the teacher-designed software with 1 8 
teachers. All the teachers received classroom sets of 
handheld computers and training in how to use the 
project’s software applications. They also participated 
in activities to show them ways to integrate the tools 
into their instruction. Activity designs were based 
in part on lessons learned from working with design 
team teachers. In addition to training, teachers had 
access to classroom-based assistance from a local 
technology coordinator and had the option of partic- 
ipating in a graduate-level course tied to the project. 
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Research and evaluation were 
part of the project in all three years 
of the grant. During the first year 
(2001-02), before designing the new 
assessment software, we interviewed 
teachers and visited their classrooms 
to learn what kinds of challenges they 
faced in science instruction and as- 
sessment. (For more on the results of 
these interviews, see my article — with 
D. Tatar and J. Roschelle — in 30 [4] 
of the Journal of Educational Com- 
puting Research) . During the second 
year (2002-03), we kept notes of the 
design teams’ process, interviewed 
teachers, and observed classroom tri- 
als with the software to learn about 
their experience with the design 
process and to identify likely effects 
of introducing the software into class- 
rooms. Finally, during the field trial 
year, we conducted a formal evalua- 
tion of the project’s effects on teachers 
and students. 

Lessons Learned 

Each lesson presented below is offered 
not as a script for success, but rather 
as a critical element to consider in 
planning a handheld initiative. 

Lesson 1: Set goals that address 
important educational problems. 

Many early 1 : 1 laptop initiatives 
focused more on increasing access 
to technology than on developing a 
clear theory for how increasing access 
would improve teaching and learning. 
In Project WHIRL, our advocacy for 
providing each student with a hand- 
held computer was driven by a focus 
on improving classroom assessment. 
We saw from earlier studies that 
handheld computers helped improve 
student engagement and that, if used 
well, their appropriate use could pro- 
vide students with opportunities for 
meaningful reflection. 

Each design team started out with 
a charter document that specified 
an educational problem their tool 
would help solve, such as the need to 


support students’ paying closer atten- 
tion to measurements in science labs. 
The resulting software proved both 
engaging to students and helpful to 
teachers. Teachers reported that the 
software engaged students much more 
than their typical pencil-and-paper 
assessments and provided them with a 
better sense of what their students un- 
derstood throughout different phases 
of instruction. They reported using 
the data to adjust instruction, either 
because most students understood 
the material or because students were 
having difficulties. 


Lesson 2: Select uses that match 
the capabilities and limitations of 
handhelds. Handheld computers 
can perform many tasks that desktop 
and laptop computers can, but it is 
a mistake to treat them as if they are 
shrunken computers. Like bigger 
computers, handhelds are useful for 
gathering, storing, and transferring 
digital information. Many software 
applications have been developed 
for them that can perform the same 
functions that productivity tools for 
desktops do, such as word processing. 
But handheld computers have some 


Software Development Planning Rubric 


Foster a development process and environment in which your partner teachers’ experience, 
wisdom, and intuition can guide, shape, and find expression in software applications that are: 

• Important to your classrooms 

• Supportive of learning goals 

• Useful in an everyday sort of way 

• Feasible to create and support 

Ask yourselves the questions below to see how your software fulfills these requirements. 

Important to Your Classrooms 

• How well can you describe the issues the software should address? 

• How important are these issues to you? 

• How many teachers within your school district might share your concern over 
these issues? 

• How many teachers outside of your school district might share your concern over 
these issues? 

Supportive of Learning Goals 

• Do the concepts supported by the software come up in many topics? 

• How central are the supported ideas to student understanding in those topics? 

• How difficult is it, currently, to test student understanding of these ideas? 

• To which aspects of hands-on science activities would you apply the software? 

• How confident are you that using this software would draw the class’s attention to the 
underlying concepts of the activity (minds-on during the course of hands-on activities)? 

• To what extent might the use of the software provide a new way of making students’ 
thinking visible to you? 

Useful in an Everyday Sort of Way 

• How often might you use the software? 

• How much prep work before class would be required to use the software? 

• How much logistical support might be necessary to use the software in class? 

• How many different steps and resources will students need to get started with a new 
software-supported activity? 

• How many different setups or versions of the software will there be? 

Feasible to Create and Support 

• Do the specific features of handheld computing match the problem? 

• How complicated is the software? 

• How important is speed of using the software? 

• How important are multimedia elements (audio, animations, video)? 

• How important are communication elements? 
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capabilities and limi- 
tations that desktops 
do not. For example, 
handheld computers 
are much more por- 
table, and the boot-up 
time is much shorter 
than for desktop 
computers. Handheld 
computers have an 
extremely small screen 
size compared to desk- 
tops, however, so it 
is difficult to portray 
complex graphical representations on 
them or to have several people look 
at the screen at the same time. When 
considering the goals of a handheld 
initiative, it is critical to consider both 
the capabilities and limitations of 
handheld computers, and to examine 
what combinations of devices and 
peripherals are likely to support those 
goals. 

To guide decisions about what 
kinds of features to develop for our 
software. Project WHIRL developed 
a rubric for each design team to use 
in creating its charter document. 

Each team needed to identify for its 
educational problem how the specific 
features of handheld computing could 
help address the problem. In some 
cases, design teams backed off imple- 
menting features that would have 
worked better with larger screens. In 
other cases, teams took advantage of 
the portability of handhelds to cre- 
ate prompts for students working in 
different corners of a science lab that 
functioned like “teachers’ assistants.” 
For example, one team created soft- 
ware that gives students feedback 
when measurements they are taking 
are implausible. Teachers who used 
that software reported that it helped 
them be in many places at once in 
a laboratory, because students who 
got feedback about their data were 
prompted to take a second look at 
measurements and discuss the feed- 
back with peers or their teacher. 


Lesson 3: Select a diverse array 
of program partners. Large-scale 
educational innovations of any kind 
need the support of policy makers 
and administrators at different levels 
of a school system. For handheld 
initiatives, it is critical to have not 
just technology leaders but also cur- 
riculum leaders on board. In the case 
of this project, BCSD’s Supervisor 
for Research, Program Evaluation, 
and Assessment was a critical partner, 
because she was directly involved in 
a wide array of assessment reform 
efforts in the district. We also devel- 
oped relationships with two software 
vendors, GoKnow and Pocket Mobil- 
ity, who provided us with discounted 
or free copies of their software pro- 
grams for participants. These relation- 
ships helped us leverage the resources 
of the grant so that we were able to 
provide a broader suite of tools for 
teachers to use. 

We learned some important les- 
sons from groups with whom we 
did not develop strong partnerships. 
We learned — too late to ensure the 
project’s sustainability beyond the 
grant period — that we also needed 
significant involvement from curricu- 
lum supervisors at the middle school 
level. Closer alignment of Project 
WHIRL with other major initia- 
tives to promote data-based decision 
making could have ensured that the 
project would continue with more 
support from a wide array of stake- 


holders within BCSD’s 
administration. We also 
might have benefited 
from a relationship with 
a hardware vendor. Al- 
though we were able to 
obtain bulk discounts 
on handheld devices 
from vendors, we en- 
countered difficulties 
obtaining adequate sup- 
port when we encoun- 
tered problems with a 
batch of handhelds we 
purchased in the project’s third year. 

Lesson 4: Focus on the classroom 
context in teacher professional 
development. Teachers need oppor- 
tunities to learn how to plan for the 
integration of handheld software into 
their instructional activities. At each 
teacher workshop, we engaged teach- 
ers in analyzing their own standards 
and curriculum for opportunities to 
integrate WHIRL software as assess- 
ment tools. We also asked teachers 
to bring in examples of student work 
generated within the software and 
engaged in discussions about what 
the work revealed about what their 
students knew and could do. Teach- 
ers reported that this aspect of their 
formal professional development 
was particularly valuable to them, 
because they got so many new ideas 
from their colleagues about how to 
integrate the use of the software into 
their instruction. They also reported 
that classroom-based assistance — pro- 
vided by the local learning technology 
coordinator — was critical in helping 
them carry out the activities they had 
planned. 

Teachers using handhelds with 
their students must also be prepared 
for the complex task of setting up 
handheld computers for use. Hand- 
held devices must be recharged peri- 
odically; if the charge is lost, student 
data can be lost as well. One way to 
save student data is to sync the data 
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with a desktop computer; 
however, many classrooms 
have just a few comput- 
ers they could use for this 
purpose. Teachers need 
opportunities to learn 
about and share strategies 
for charging, syncing, and 
storing handheld comput- 
ers in the classroom. In 
Project WHIRL, we con- 
structed special charging 
stations that allowed all 
handhelds in a classroom 
set to be charged all at once. Instead 
of syncing each device to a desktop, 
teachers designated one handheld as a 
collector device. This device served as a 
repository for student work that was 
to be saved. 

Handheld computers also require 
planning for different kinds of class- 
room communication than are typi- 
cal. For example, teacher-to-student 
communication was handled by using 
an infrared beam to send both soft- 
ware and student assessments with 
a “cascading” method. The teacher 
beamed an assessment to two stu- 
dents, who then each beamed it to 
two more students, and so forth, until 
the entire class had the assignment. 
Student-to-student communication 
was also supported by this peer beam- 
ing strategy, and student-to-teacher 
communication involved students’ 
beaming their work to the collector 
device, which was then synced to the 
teacher’s own desktop computer to 
preserve student work. Teacher-to- 
class communication was facilitated 
by document cameras attached to a 
video display; these allowed a teacher 
to show the entire class work from the 
collector device and then discuss the 
work of the entire class. 

Lesson 5: Use data from early 
users to decide how to evaluate 
success. A critical task for researchers 
is to help initiative leaders decide how 
to evaluate their success. Although 
this task may seem straightforward, in 


fact handheld computers are so new 
that we understand very little about 
how they change instruction, alter 
communication patterns, or influence 
student learning. 

A good strategy is to rely on class- 
room observations and interviews 
with teachers who are involved early 
in the project, and then to analyze 
the data collaboratively to identify 
the most visible effects of introduc- 
ing handhelds to the classroom. SRI 
researchers worked closely with de- 
sign team teachers in the second year 
of the project as they implemented 
prototypes of the software with their 
students, and then the design teams 
analyzed the results of those trials 
in group meetings. Those analyses 
helped inform the design of the soft- 
ware, and they also led us to form 
some hypotheses of specific effects 
on instruction we expected to see 
in the third year. We then designed 
interviews, observation protocols, 
and survey instruments to test those 
hypotheses. The advantage of adopt- 
ing this approach was that our instru- 
ments proved sensitive to the kinds of 
effects the initiative had on teachers 
and students. 

Expectations 

Two of the preliminary outcomes we 
have observed from Project WHIRL 
are changes in teachers’ thinking 
about assessment and increases in the 
level of teachers’ technology integra- 


tion into science teach- 
ing. Most of the teach- 
ers in the project were 
more likely to report 
that they used class- 
room assessment data to 
adjust their instruction. 
They also reported that 
they were able to assess 
students more frequent- 
ly because the handheld 
software programs saved 
them time. The teachers 
who began the project 
in 2002-03 were, by and large, infre- 
quent users of technology with their 
students. But by the end of Project 
WHIRL, technology use had become 
a frequent, integral part of instruction 
for most of the teachers. 

This last outcome — better integra- 
tion of technology in the curricu- 
lum — may be a secondary goal that is 
possible for many handheld initiatives 
to achieve, even if an initiative is not 
focused on improving classroom as- 
sessment. In this respect, one of the 
key promises of handheld comput- 
ers — to increase dramatically students’ 
access to technology — can be realized. 
But an even more important goal — 
increasing students’ opportunity to 
learn with technology — may also be 
achieved, although the details will dif- 
fer, depending on the educational fo- 
cus of the handheld initiative. Success 
in achieving this final goal depends 
on the match of the initiative’s focus 
with the capabilities and limitations 
of handheld computers and on the 
support given to teachers who must 
implement the program. 

• J* [I William R. Penuel is senior 

^ ly Technology in Learning at 
* SRI International. His research 

¥1 focuses on assessment and 

If IJ evaluation of large-scale tech- 
nology supported reforms in 
schools and community settings. He is principal 
investigator for Project WHIRL. He holds a doc- 
torate in developmental psychology from Clark 
University. 


10 


Learnins & Leadins with Technolosy Volume 32 Number 6 

Copyright © 2005, ISTE (International Society forTechnology in Education), 1. 800. 336.5 19 1 (U.S. & Canada) or 1 .54 1 .302.3777 (Inti), iste@iste.org, www.iste.org.AII rights reserved. 




